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Abstract 


Acquisitions  of  software  intensive  systems  by  the  Department  of  Defense  (DoD)  have  often 
suffered  from  poor  product  quality,  cost  overruns,  and  schedule  slips.  In  turn,  these  problems 
have  frequently  been  linked  to  the  inability  of  project  offices  to  successfully  manage  the  acqui¬ 
sition  of  the  software  components  of  the  systems. 

There  have  been  a  number  of  efforts  to  provide  the  necessary  education  and  training  to 
improve  the  skills  and  capabilities  of  managers  for  software  intensive  acquisitions.  However, 
acquisition  problems  remain  pervasive  in  the  DoD  [Cavanaugh  98]. 

More  must  be  known  about  the  causes  and  underlying  issues  surrounding  these  problems. 
Specifically,  the  needs  of  the  acquisition  management  offices  must  be  better  understood  to 
help  them  improve.  This  includes  a  better  understanding  of  how  education  and  training  can 
improve  the  individual  manager’s  skills  and  competency  related  to  acquiring  such  systems. 

To  elicit  these  needs,  the  Software  Engineering  Institute  (SEI)  conducted  a  survey  of  senior 
acquisition  managers.  The  survey  focused  on  the  performance  of  their  organizations,  particu¬ 
larly  with  respect  to  a  series  of  skills  and  competency  areas  that  may  affect  an  organization’s 
ability  to  successfully  acquire  software  intensive  systems. 

Results  indicate  that  the  program  executive  officers  (PEOs)  and  program  managers  (PMs)  who 
completed  the  survey  were  reasonably  well  satisfied  with  the  capabilities  of  their  organiza¬ 
tions  to  acquire  software  intensive  systems.  In  many  cases,  however,  the  source  of  the  exper¬ 
tise  for  such  acquisitions  were  contractors  either  supporting  the  organizations  or  the  prime 
contractors  developing  these  systems.  Comparable  expertise  often  was  unavailable  in  govern¬ 
ment  acquisition  organizations  themselves.  From  this  fact,  the  need  for  government  expertise 
in  these  acquisitions  was  noted.  In  addition,  the  survey  queried  participants  on  the  best  way  to 
obtain  this  expertise  through  education  and  training. 

Finally,  recommendations  derived  from  survey  results  are  offered  to  increase  software  acquisi¬ 
tion  education  and  training  opportunities  for  managers. 
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1  Introduction 


A  great  many  systems  being  acquired  by  the  Department  of  Defense  (DoD)  are  heavily 
dependent  upon  software.  These  systems  include  automated  information  systems  (AIS), 
weapon  systems  (WS),  and  command,  control,  and  communication,  intelligence  electronic 
warfare  systems  (C3IEWS). 

Acquisitions  of  such  systems  often  suffer  from  continued  failure  of  the  acquisition  and 
development  efforts  to  meet  cost,  schedule,  and  performance  goals.  These  difficulties  have 
been  linked  to  the  inability  of  both  the  acquirer  and  the  developer  to  manage  the  acquisition 
process,  and  the  developer  to  manage  the  development  process,  especially  where  software  is 
involved  [OUSD  94]. 

More  than  three  years  ago,  Secretary  of  Defense  Cohen’s  noted  in  a  press  conference  initiating 
Acquisition  Reform  Week  [Cohen  97]: 

The  challenge  is  really  to  apply  new  practices  to  all  of  our  programs  across  the 
board — large  and  small.  And  we  have  to  make  acquisition  reform  a  part  of  our 
everyday  life.  And  we  have  to  continue  to  develop  an  acquisition  workforce,  and 
that’s  also  a  challenge  because  they  need  to  have  the  skills  and  tools  along  with 
the  motivation. 

However,  the  specific  approaches  that  could  result  in  successful  acquisitions  have  not  always 
been  clearly  identified  or  implemented  [Cavanaugh  98].  More  needs  to  be  known  about  the 
issues  and  causes  that  can  explain  varying  success  in  acquisition.  Specifically,  the  needs  of  the 
acquisition  management  offices  must  be  better  understood  to  help  them  improve  their 
acquisitions  of  software  intensive  systems.  This  includes  a  better  understanding  of  how 
education  and  training  can  improve  the  individual  manager’s  skills  and  competency  in 
acquiring  such  systems. 

To  elicit  these  needs,  the  Software  Engineering  Institute  (SEI)  conducted  a  survey  of  senior 
acquisition  managers.  The  survey  focused  on  the  performance  of  their  organizations, 
especially  on  needed  skills  and  competencies,  and  on  issues  surrounding  the  training  needed  to 
develop  them  in  both  the  project  office  staffs  and  for  the  senior  acquisition  managers 
themselves. 
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We  discuss  the  results  of  the  survey  in  this  report.1  While  the  specific  purpose  was  to  better 
understand  the  needs  of  project  offices,  the  results  also  have  wider  import  with  respect  to 
organizational  process  improvement  and  acquisition  reform.  Section  2  briefly  describes  the 
survey.  Section  3  presents  the  results  in  graphical  form.  In  addition.  Section  3  explores  some 
general  themes  common  to  these  participants’  responses.  In  Section  4  we  discuss  general 
observations  we  derived  from  the  survey  results.  Based  on  these  observations,  we  give  some 
recommendations  to  help  managers  obtain  needed  software  acquisition  expertise. 


1.  Many  people  have  contributed  to  the  successful  completion  of  this  effort.  In  particular,  the 
authors  wish  to  thank  Sally  Cunningham  Jon  Gross,  Bob  Lang,  Bill  Peterson,  Scott  Reed,  Bob 
Rosenstein,  Sheila  Rosenthal,  and  Dave  Zubrow.  Of  course,  special  thanks  also  are  due  to  the 
many  senior  acquisition  personnel  who  took  the  time  out  of  their  busy  schedules  to  provide  the 
information  that  made  this  report  possible. 
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2  Approach 


This  report  is  based  on  a  survey  of  senior  acquisition  personnel,  using  a  structured,  self- 
administered  questionnaire  that  was  available  both  electronically  via  the  World  Wide  Web  and 
in  paper  form  (see  Appendix).  The  questions  were  phrased  in  both  pre-coded  “closed  ended” 
and  free  form  “open  ended”  format,  allowing  the  participants  to  more  fully  elaborate  on  their 
responses.  There  was  also  space  for  additional  written  remarks  and  suggestions  for 
improvements. 

The  questionnaire  was  structured  into  four  main  question  sets.  “Your  Background  in  the 
Acquisition  of  Software  Intensive  Systems”  includes  questions  about  organizational  roles, 
type  of  systems  acquired,  previous  experience  and  training  in  software  acquisition,  and 
personal  expertise  in  software.  “About  Your  Acquisition  Organization”  includes  a  series  of 
questions  about  performance  in  32  areas  of  organizational  competency.  ‘Training  Issues”  asks 
about  quality  of  software  education  and  training,  the  need  for  additional  preparation  for 
software  acquisitions  management,  and  delivery  methods  for  training.  Finally,  in  “Problems 
Faced  in  Acquiring  Software  Intensive  Systems,”  we  asked  two  overall  summary  questions 
about  particularly  difficult  problems  and  recommendations  for  improving  the  acquisition  of 
such  systems. 

The  closed  ended  responses  were  summarized  with  simple  descriptive  statistics  in  text  and 
graphical  form.  The  open  ended  responses  were  classified  by  recurring  themes,  and  both  sets 
of  results  were  compared  for  consistency. 

The  survey  was  administered  during  the  Summer  of  1998.  A  total  of  81  senior  acquisition 
personnel,  approximately  60  percent  of  those  initially  queried,  completed  and  returned  their 
questionnaires. 
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3  Results 


3.1  Participant  Background 

The  survey  participants  were  considered  to  be  senior- level  managers  (Figure  1).  Over  forty 
percent  were  Program  Executive  Officers  (PEOs),  with  the  remainder  spread  among  Program, 
Project,  and  Product  Managers  (PMs)  and  their  deputies.  All  have  had  considerable 
responsibilities  for  the  acquisition  of  a  variety  of  software  intensive  systems.  As  seen  in 
Figure  2,  the  large  majority  (over  70%)  have  participated  in  acquisitions  of  weapons  systems, 
while  well  over  a  third  have  acquired  automated  information  systems.  Similarly,  well  over  a 
third  have  acquired  C3EEW  systems. 


N  =  77 

Figure  1:  Organizational  Roles 


SPEO 

■  Program  Manager 

■  Project  Manager 
□  Product  Manager 


Summary  based 

on  sample 
selection  criteria. 


Percent 
so  -r 


Automated  Weapons  C3IEW  Other 
Information  Systems 
n  *  si  Systems 


Summary  of  responses  to  question  1 
on  page  2  of  the  questionnaire 
reproduced  in  the  Appendix. 


Figure  2:  Systems  Acquired  by  Respondents 
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3.2  Organizational  Performance 

We  asked  the  survey  participants  a  broadly  stated  question  about  the  success  of  their 
organizations  in  acquiring  the  software  for  their  systems.  Most  PEOs  and  PMs  rated  their 
organizations  reasonably  high  in  their  performance  in  meeting  schedule  and  budget 
commitments,  and  the  technical  performance  of  the  software  once  deployed.  Well  over  a  third 
of  them  characterized  their  organizations’  overall  performance  as  excellent  or  even 
exceptional  (Figure  3). 

That  said,  the  plurality  (over  40%)  did  choose  the  middle  category  of  “good”  performance. 
And  a  noticeable  minority  (another  18%)  rated  their  organizations  as  performing  adequately  at 
best.2 


Figure  3:  Performance  in  Acquiring  Software 


3.3  Organizational  Capability 

We  asked  the  survey  participants  to  rate  the  capabilities  of  their  organizations  in  each  of  a 
series  of  22  key  competency  areas  that  have  been  identified  by  experts  as  being  crucial  for  the 
successful  acquisition  of  software  intensive  systems  [Cavanaugh  98].  The  responses  cannot 
necessarily  be  taken  as  direct  measures  of  “need”  for  improvement,  but  they  do  identify  those 
areas  where  there  may  be  an  audience  receptive  to  change. 

The  key  competency  areas  are  broken  down  in  Figures  4  through  7  according  to  the  rank  order 
in  which  the  participants  rated  the  capability  of  their  acquisition  organizations.  The  areas  are 
ranked  from  highest  to  lowest  incidence  of  “poor”  and  “adequate”  responses  (with 


2.  There  are  “social  desirability”  effects  in  these  kinds  of  survey  data.  For  example,  different  ratings 
might  be  forthcoming  from  subordinate  staff.  In  fact,  the  PEOs  in  the  current  study  were  some¬ 
what  more  likely  to  say  their  organizations  did  an  exceptional  or  excellent  job  (46%)  than  were 
the  PMs  (36%). 
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“unnecessary”  indicating  the  least  need  for  improvement).  Figure  4  summarizes  the  areas 
asserted  to  be  relatively  most  in  need  of  improvement.  Figure  7  includes  the  competency  areas 
deemed  by  the  survey  participants  to  be  least  problematic. 

The  first  thing  to  notice  in  the  four  figures  is  that  these  PEOs  and  PMs  generally  gave  their 
organizations  relatively  high  competency  ratings.  However,  their  responses  did  vary  by 
competency  area,  and  “good”  (i.e.,  less  than  “excellent”)  is  almost  always  the  modal  category. 
That  pattern  of  responses  typically  indicates  room  for  improvement. 


Notice  in  Figure  4  that  the  largest  number  of  respondents  recognized  that  their  organizations 
had  difficulty  in  providing  appropriate  training  to  their  personnel.  Just  over  50%  of  these 
PEOs  and  PMs  characterized  the  training  opportunities  provided  by  their  organizations  as 
having  been  poor  or  adequate  at  best.  Risk  management  came  next,  with  well  over  40% 
expressing  similar  reservations  about  their  organizations’  capabilities  in  that  area.  Cost  and 
schedule  estimation  were  cited  similarly  by  essentially  the  same  number. 


Ns  80 


Training  RiskMgt  Cost/Sched.  Software  Reuse  Software  SE  Approaches 
Provided  Estimation  Quality  Mgt  and 

(M=79)  Methodologies 

B  Unnecessary  B  Exceptional  1  Excellent  □  Good  El  Adequate  ■  Poor 


Summary  of  responses  to  questions  1.21, 1.17, 
1.5, 1.16, 1.13, 1.1,  respectively,  on  pages  4  and  5 
of  the  questionnaire  reproduced  in  the  Appendix. 


Figure  4:  Key  Competency  Areas 


The  next  most  problematic  group  of  key  competency  areas  is  summarized  in  Figure  5.  Over 
one  quarter  of  the  respondents  characterized  their  organizations’  capabilities  in  these  areas  as 
being  adequate  or  worse.  Note  that  the  response  pattern  for  understanding  “emerging  software 
issues  and  technologies”  closely  follows  response  pattern  for  the  similar  category  in  Figure  4 
of  understanding  the  latest  “software  engineering  approaches  and  methodologies.”  Along 
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Software  Assessing  Software 

Security  Process  Requiremnts 
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Summary  of  responses  to  questions  1.15, 1.11, 1.19, 
1.18,  and  1.7,  and  1.14,  respectively,  on  pages  4  and 
5  of  the  questionnaire  reproduced  in  the  Appendix. 


Figure  5:  Key  Competency  Areas 


with  software  reuse  (also  in  Figure  4)  and  security,  many  of  these  PEOs  and  PMs  recognized 
the  importance  of  keeping  up  with  changing  technical  trends. 


Not  surprisingly,  requirements  management  makes  the  list  of  areas  cited  by  over  one  fourth  of 
these  PEOs  and  PMs  as  having  been  adequate  at  best  in  their  organizations.  As  we  will  see 
below,  requirements  issues  were  frequently  cited  as  among  the  most  difficult  problems  faced 
by  acquisition  organizations. 


Note  also  that  several  measurement  and  evaluation  related  categories  were  among  those  key 
competency  areas  that  were  cited  most  often  as  having  been  adequate  at  best.  In  addition  to 
“cost  and  schedule  estimation”  and  “software  quality  management”  from  Figure  4,  these 
include  “software  reviews  and  audits,”  “using  software  metrics,”  “assessing  process  maturity,” 
and  aspects  of  “software  requirements  management.”  Figures  6  and  7  summarize  those  areas 
that  PEOs  and  PMs  reported  as  being  in  need  of  relatively  less  improvement.  In  fact,  the 
respondents  rated  their  organization’s  performance  for  these  areas  as  adequate  to  poor  in  less 
than  20%  of  the  cases. 
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Summary  of  responses  to  questions  1 .22,  1 .20, 1 .6, 
1 .3,  and  1 .2,  respectively,  on  pages  4  and  5  of  the 
questionnaire  reproduced  in  the  Appendix. 


Figure  6:  Key  Competency  Areas 
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3.4 


Levels  of  Expertise 


3.4.1  Previous  Experience 

Our  respondents  reported  that  their  previous  experience  and  training  had  prepared  them 
reasonably  well  for  their  then  current  work  in  the  acquisition  of  software  intensive  systems 
(Figure  8).  Almost  one  half  of  them  characterized  themselves  as  having  been  extremely  or 
very  well  prepared.  However,  the  largest  group  (over  40%)  said  they  were  only  moderately 
well  prepared  for  their  current  duties,  and  just  over  10%  contended  that  they  were  not  very 
well  prepared. 

Most  of  the  PEOs  and  PMs  who  participated  in  the  study  said  that  they  personally  had  a 
substantial  amount  of  expertise  in  the  acquisition  of  software  intensive  systems.  However, 
they  had  considerably  less  confidence  in  their  expertise  with  software  acquisition  and  the 
technical  aspects  of  software  engineering  (Figure  9).  Almost  two  thirds  of  them  characterized 
themselves  as  having  substantial  or  even  extensive  expertise  in  managing  software  intensive 
system  acquisitions.  But  over  half  told  us  that  they  had  only  moderate  or  little  personal 
expertise  in  software  acquisition,  and  over  70%  said  that  they  had  comparably  little  expertise 
in  the  technical  aspects  of  software  engineering. 


How  well  did  it  prepare  them 
for  software  acquisition? 


■  Extremely  Well 

■  Very  Well 

■  Moderately  Well 
□  Not  Very  Well 


Summary  of  responses  to  question 
2.7  on  page  3  of  the  questionnaire 
reproduced  in  the  Appendix. 


Figure  8:  Previous  Experience  and  Training 


3.4.2  Expertise  Needed  by  Senior  Management 

Similar  to  their  own  personal  backgrounds,  most  of  our  respondents  suggested  that  senior 
management  needs  a  broad  understanding  of  software  issues  rather  than  detailed  technical 
expertise.  The  vast  majority  of  the  survey  participants  said  that  senior  management  generally 
needs  a  broad,  rather  than  detailed,  understanding  of  technical  issues  in  software  engineering 
as  well  as  of  software  and  system  acquisition  (Figure  10).  However,  over  a  third  also  argued 
that  senior  managers  need  detailed  technical  expertise  in  software  and  system  acquisition. 
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EJ  Extensive 

■  Substantia! 

■  Moderate 
B  Little  if  any 


N  =  81 


Technical 
Aspects  of 
Software 
Engineering 


Management  of  Management  of 
Software  Software 

Acquisitions  Intensive 

System 
Acquisitions 


Summary  of  responses  to  questions 
3.1 , 3.2,  and  3.3  on  page  3  of  the 
questionnaire  reproduced  in  the 
Appendix. 


Figure  9:  Personal  Expertise  in  Software 


And  almost  twenty  percent  suggested  that  detailed  technical  knowledge  is  necessary  in  the 
technical  aspects  of  software  engineering  as  well. 

3.4.3  Sources  of  Software  Expertise 

Concerns  are  often  expressed  that  program  offices  frequently  do  not  have  sufficient,  unbiased 
expertise  available  to  them  to  support  the  management  of  the  software  related  aspects  of  their 
acquisitions.  This  is  so  particularly  when  they  rely  primarily  on  the  industry  contractors  from 
whom  their  organizations  acquire  their  systems.  However,  other  organizations  do  have 
comparable  expertise  available  from  their  own  in-house  staff,  from  other  government  support 
organizations,  or  from  other  contractors  who  provide  direct  support  to  their  organizations  for 
managing  systems  that  are  acquired  elsewhere. 

We  asked  the  PEOs  and  PMs  on  whom  they  relied  for  expertise  about  the  software-related 
aspects  of  their  system  acquisitions.  About  two  thirds  of  them  said  that  they  relied  to  a  large 
extent  or  even  almost  entirely  on  the  contractors  from  whom  they  acquired  their  software 
intensive  systems  (Figure  11).  However,  about  half  of  them  placed  similar  reliance  on  their 
own  in-house  staff,  nearly  as  many  relied  on  other,  independent  contractors,  and  over  40% 
reported  that  they  depended  similarly  on  government  support  organizations. 

Not  surprisingly,  many  program  offices  rely  on  more  than  one  source  of  expertise.  As  also 
seen  in  Figure  11,  very  large  majorities  of  the  PEOs  and  PMs  in  our  survey  said  that  their 
organizations  relied  at  least  to  some  extent  on  all  four  of  these  potential  sources  of  guidance. 
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N  =  80 


Technical 
Aspects  of 
Software 
Engineering 


Management 
of  Software 
Acquisitions 

(N  ■  79) 


Management 
of  Software 
Intensive 
System 
Acquisitions 


□  Detailed  Technical 

■  Both  Detail  &  Breadth 

■  Broad  Understanding 

■  Neither 


Summary  of  responses  to  questions 
4,1 , 4.2,  and  4.3  on  page  6  of  the 
questionnaire  reproduced  in  the 
Appendix. 


Figure  10:  Software  Expertise  Needed  by  Senior  Management 


100% 


80% 


60% 


40%  - 


20% 


N  =  79 


TjH  g  jjg  _ 


0% 


I 


1 1 1 1 

Acquisition  In-house  Staff  Support  Government 
Contractors  Contractors  Support 


□  Almost  entirely 
■  To  a  large  extent| 
i  To  some  extent 


l  Hardly  at  all 


Organizations 


Summary  of  responses  to 
questions  3.1 ,  3.2,  3,3,  and  3.4  on 
page  6  of  the  questionnaire 
reproduced  in  the  Appendix. 


Figure  1 1:  Sources  of  Software  Expertise 


Of  course  it  makes  a  great  deal  of  sense  for  organizations  to  seek  guidance  and  corroboration 
from  more  than  one  source  of  expertise.  Indeed,  there  is  nothing  wrong,  and  much  right,  with 
expecting  pertinent  information  and  expertise  from  competent  contractors.  The  concern  is 
with  undue  reliance  on  biased  or  otherwise  limited  sources  of  information  without 
corroboration. 

In  fact,  only  17%  of  these  PEOs  and  PMs  reported  relying  to  a  large  extent  or  more  on  their 
suppliers  exclusively  for  guidance  about  managing  the  software  related  aspects  of  their  system 
acquisitions  (Figure  1 2).  However,  another  25%  did  say  that  they  relied  to  a  similar  extent  on 
a  second  source  from  the  other  3  sources  of  expertise,  in  addition  to  their  suppliers.  Another 
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17%  relied  extensively  on  only  a  single  one  of  the  3  sources  of  expertise  other  than  their 
suppliers.  But  that  leaves  40%  who  said  that  they  in  fact  placed  similar  reliance  on  2  or  all  3 
sources  of  expertise  other  than  their  suppliers.  Seventeen  percent  said  they  rely  heavily  on  two 
sources  of  expertise  not  including  their  suppliers.  Fifteen  percent  relied  on  two  other  sources 
in  addition  to  their  suppliers.  And  9%  said  they  relied  on  all  four  sources  of  expertise 
including  their  suppliers. 


BD  Acquisition  Contractor 
Only 

■  Acquisition  Contractor 
Plus  One  Other 

■  One  Non  Acquisition 
Contractor  Only 

□  Two  or  More  Non 
Acquisition  Contractors 


Summary  of  responses  to  question 
3  on  page  6  of  the  questionnaire 
reproduced  in  the  Appendix. 


Figure  12:  Multiple  Sources  of  Expertise 


Moreover,  there  is  at  least  some  evidence  from  these  data  that  relying  too  heavily  on 
acquisition  contractors,  and/or  any  other  single  source  of  information  and  expertise,  did  seem 
to  adversely  affect  overall  performance  in  acquiring  the  software  for  these  systems  (Figure 
1 3).  Fewer  than  a  third  (32%)  of  those  who  said  that  they  relied  exclusively  on  their  suppliers 
and/or  one  other  source  of  expertise  other  than  their  suppliers  characterized  their 
organizations’  overall  performance  as  being  excellent  or  exceptional.  However,  almost  one 
half  of  those  who  relied  on  multiple  sources  of  software  related  expertise  also  reported  that 
their  organizations  had  had  excellent  or  even  exceptional  success  in  their  software 
acquisitions. 


3.5  Training  Issues 

In  this  section  we  briefly  review  the  survey  participants’  judgments  about  the  need  for 
improvement  in,  and  the  difficulties  they  faced  in  improving,  education  and  training  for 
software  intensive  systems. 


3.5.1  Quality  of  Software  Education  and  Training 

The  PEOs  and  PMs  who  completed  our  survey  saw  substantial  room  for  improvement  in  then- 
current  DoD  education  and  training  with  respect  to  software  acquisitions. 
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Exceptional  or  Excellent 
Performance  Reported 

100% 

80% 

60% 

40% 

20% 

0% 


[  Crosstabulation  of  responses  to  question  2  and  3  on  page  6  of  the  questionnaire  reproduced  in  the  Appendix. 

Figure  13:  Performance  in  Acquiring  Software  by  Sources  of  Software  Expertise 


Heavy  Reliance  on  Acquisition  Heavy  Reliance  on  Two  or  More  Non 
Contractors  and/or  Another  Single  Acquisition  Contractors  (N  =  33) 
Source  of  Expertise  (N  =  48) 


DoD  Software  DoD  "How  To"  Software 
Acquisition  Skills 

N  =  eo  Management 


ES  Exceptional 
□  Excellent 

■  Good 

■  Adequate 

■  Poor 


Summary  of  responses  to  questions  1 
and  2  on  page  7  of  the  questionnaire 
reproduced  in  the  Appendix, 


Figure  14:  Quality  of  Software  Education  and  Training 


As  seen  in  Figure  14,  more  than  half  of  the  survey  respondents  characterized  as  good  or  better 
the  current  DoD  education  and  training  courses  in  preparing  their  organizations  for  managing 
the  software  related  aspects  of  their  acquisitions.  However,  a  large  minority  (46%)  said  it  was 
adequate  or  worse,  while  a  much  smaller  group  (16%)  asserted  that  it  was  excellent  or 
exceptional. 


When  asked  a  more  pointed  question  about  how  well  then-current  DoD  education  and  training 
conveyed  software  related  skills  with  sufficient  detail  and  direction  about  how  to  perform 
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necessary  tasks,  the  majority  (over  60%)  said  that  the  situation  was  only  adequate  at  best. 
Fewer  than  5%  characterized  it  as  excellent  or  exceptional. 


Indeed,  a  majority  of  the  respondents  said  there  “should  be  a  separately  defined  career  path  for 
specialists  in  software  acquisition  management”  (Figure  15).  The  vast  majority  (over  90%) 
agreed  that  “courses  or  comparable  experience  in  software  related  management  and  technical 
topics  [should]  be  required  for  all  members  of  the  acquisition  corps  who  work  on  the 
acquisition  of  software  intensive  systems.” 


Figure  15:  Need  for  Additional  Preparation  for  Software  Intensive  Acquisitions 


3.5.2  Training  Delivery  Mechanisms 

Various  “distance  learning”  methods  and  approaches  have  been  discussed  in  recent  years  as 
being  cost  effective  alternatives  to  traditional  classroom  instruction,  especially  for  the  delivery 
of  technical  training.  There  is  also  a  recurrent  concern  in  education  circles  with  the 
appropriateness  of  pre-service  versus  in-service  training. 

We  asked  our  participating  PEOs  and  PMs  to  identify  which  methods  their  organizations 
currently  relied  on  “to  prepare  personnel  to  manage  the  acquisition  of  software  intensive 
systems.”  We  then  asked  them  to  contrast  the  current  situation  with  what  delivery 
mechanisms  they  ought  to  use.  Their  responses  are  summarized  in  Figures  16  through  22. 

First  of  all,  notice  that  the  survey  respondents  said  that  they  would  prefer  to  rely  more  heavily 
than  they  currently  did  on  pre-employment  training  (Figure  16).  Almost  two  thirds  said  that 
they  currently  relied  at  least  “frequently”  on  formal  education  and  pre-employment  training. 
However,  almost  90%  said  they  ought  to  do  so,  with  almost  half  saying  they  should  “almost 
always”  rely  on  such  prior  preparation. 


CMU/SEI-2000-TR-003 


15 


100% 

80% 

60% 

40% 

20% 

0% 


29 


49 


Currently  Ought  to ... 

<N-79>  ^(N-77) 


□  Almost  always 

■  Frequently 

■  Occasionally 

■  Rarely  if  ever 


Summary  of  responses  to  the 
question  5  series  on  page  8  of  the 
questionnaire  reproduced  in  the 
Appendix. 


Figure  16:  Training  Delivery  Systems:  Formal  Education  and  Pre-Employment 


That  said,  many  of  the  respondents  stated  they  would  like  to  rely  more  heavily  on  in-service 
training  and  continuing  education  as  well  (Figure  1 7).  While  a  majority  said  they  currently 
used  continuing  education  and  training  to  prepare  their  acquisition  personnel,  fewer  did  so 
than  relied  on  pre-employment  education  and  training.  However,  many  of  them  would  have 
liked  to  offer  more  in-service  training  and  continuing  education  opportunities  than  they  then 
were  able  to  provide.  Well  over  90%  said  that  they  ought  to  provide  such  opportunities  at  least 
frequently. 


Figure  1 7:  Training  Delivery  Systems:  In-Service  and  Continuing  Education 


Actually,  more  of  the  survey  respondents  reported  that  their  organizations  currently  relied  on 
on-the-job  and  developmental  assignments  than  on  continuing  education  and  in-service 
training  (Figure  18). 


16 


CMU/SEI-2000-TR-003 


However,  there  is  little  change  in  the  number  of  PEOs  and  PMs  who  said  they  wanted  to  take 
more  advantage  of  developmental  assignments  than  they  currently  relied  upon. 


Figure  18:  Training  and  Delivery  Systems:  On  the  Job  Training  and  Developmental 
Assignments 


The  change  between  current  and  desired  state  is  most  pronounced  for  distance  learning 
methods  that  we  characterized  as  “Web  based,  CD  ROM’s,  multimedia,  satellite  broadcasts, 
network  and  computer-based  instruction,  collaborative  groupware.”  Fewer  than  10%  of  the 
PEOs  and  PMs  we  surveyed  reported  using  such  methods  frequently  or  more  often  (Figure 
19).  However,  more  than  half  said  that  they  ought  to  do  so  at  least  frequently.  Almost  all  of 
them  said  that  they  ought  to  try  distance  learning  methods  at  least  occasionally. 


Figure  19:  Distance  Learning  Methods 
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Figure  21:  Training  and  Delivery  Systems:  Focused  “Just-in-Time”  Training 


Similarly,  the  PEOs  and  PMs  were  considering  greater  use  of  “just  in  time”  (JIT)  training  to 
keep  their  personnel  up  to  date  with  rapid  changes  in  the  field  (Figure  21).  Not  unlike  the 
situation  with  distance  learning  methods,  though,  many  of  the  respondents  still  appeared 
tentative  in  their  judgments  and  were  considering  only  occasional  use  of  JIT  training. 


Figure  20:  Training  and  Delivery  Systems:  Commercial  “Off-the-ShelF  Training 
Courses 


Commercial  off-the-shelf  (COTS)  courses  are  becoming  increasingly  available  in  many  areas 
related  to  information  technology.  We  asked  about  the  use  of  such  alternatives  “to  augment  or 
replace  training  provided  directly  by  the  DoD  and  other  government  agencies.”  As  with  most 
of  the  methods  about  which  we  asked,  there  was  some  increased  interest  in  COTS  courses, 
although  with  less  marked  change  than  was  so  for  distance  learning  and  JIT  (Figure  20). 
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Finally,  we  asked  about  existing  DoD  courses.  While  there  was  interest  in  distance  learning 
and  other  alternative  delivery  mechanisms,  these  PEOs  and  PMs  clearly  intended  to  continue 
to  rely  on  DoD  course  offerings  (Figure  22). 


100% 


60% 


40% 


20% 


0% 


Currently  Ought  to ... 

(M  *7f)  (N  *76) 


□  Almost  alwways 

■  Frequently 

■  Occasionally 

■  Rarely  if  ever 


Summary  of  responses  to  the 
question  5  series  on  page  8  of  the 
questionnaire  reproduced  in  the 
Appendix. 


Figure  22:  Training  Delivery  Systems:  Existing  DoD  Courses 


3.5.3  Limitations  on  Adequate  Training 

We  showed  the  survey  respondents  a  series  of  statements  about  limitations  on  an 
organization’s  ability  to  provide  software  related  training  to  its  acquisition  personnel,  and  then 
asked  them  how  well  the  statements  applied  to  their  organizations.  Their  responses  are 
summarized  in  Figures  23  and  24,  in  rank  order  from  the  most  to  least  frequently  reported 
problems. 

Note  the  most  commonly  reported  problem  was  that  defined  organizational  processes  too  often 
were  abandoned  during  critical  events  and  crises.  Over  40%  of  the  PEOs  and  PMs  surveyed 
said  that  such  problems  adversely  affected  the  ability  of  their  organizations  to  provide 
software  related  training  “to  a  large  extent”  or  even  “almost  entirely.”  Almost  90%  said  that 
such  described  their  organizations  at  least  “to  some  extent.”  In  a  related  vein,  note  also  the 
emphasis  in  the  remainder  of  Figure  24  on  not  having  enough  time  or  resources  available  to 
devote  to  training. 

Finally,  notice  in  Figure  24  that  the  PEOs  and  PMs  often  reported  that  not  enough  good 
software  related  courses  currently  were  available  for  use  by  their  organizations.  Only  about 
20%  of  them  said  that  such  problems  accurately  described  the  situation  in  their  organizations 
at  least  to  a  large  extent  or  more,  but  over  three  quarters  said  their  organizations  were  affected 
by  such  problems  at  least  to  some  extent. 
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Overcome  by  Time  to  Send  Time  to  Bring  Too  Long  &  Rotate  Out  at  Staff  to 
Crises  People  Off  Site  New  Staff  Up  Demanding  Inopportune  Downsizing 
to  Speed  Times  <n  « 80) 

|N»IO)  (N  .  M) 

■  Hardly  at  all  1  To  some  extent  ■  To  a  large  extent  □  Almost  entirely 

Summary  of  responses  to  questions  6,11 , 6.9, 
6.10,  6.3,  6.1,  and  6.8  on  page  8  of  the 
questionnaire  reproduced  in  the  Appendix. 


Figure  23:  Limitations  of  Providing  Adequate  Training 
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■  Hardly  at  all  ■  To  some  extent  ■  To  a  large  extent  □  Almost  entirely 


Summary  of  responses  to  questions  6.7,  6.2,  6.4, 
6.5,  and  6.6  on  page  8  of  the  questionnaire 
reproduced  in  the  Appendix. 


Figure  24:  Limitations  of  Providing  Adequate  Training 
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3.6  Problems  Faced  in  Acquiring  Software  Intensive 
Systems 

In  addition  to  the  closed  ended  questions  concerning  key  competency  areas,  we  also  asked  the 
survey  participants  two  broad,  open-ended  questions.  After  asking  them  to  describe  the  one 
or  two  most  difficult  problems  that  their  organizations  have  faced  in  conducting  successful 
acquisitions,  we  asked  them  to  consider  the  one  or  two  things  they  would  most  like  to  see 
changed  in  how  software  intensive  systems  are  acquired  in  the  DoD. 

A  variety  of  issues  surfaced  in  the  responses  to  these  two  questions.  However,  three  clear 
themes  emerged  as  particularly  significant. 


3.6.1  Requirements  Development  and  Management 

The  need  for  better  knowledge  in  the  development  and  management  of  system  and  associated 
software  requirements  is  evident  from  our  respondents’  free  form  answers.  The  following 
aggregated  quotes  highlight  the  concerns  of  the  respondents. 

The  most  significant  problems  are  associated  with  the  following: 

•  requirements  creep 

•  change  of  requirements  because  of 

customer/user  constantly  redefining  and  dictating  requirements 
the  effect  of  technology  changes 
the  effect  of  budget  volatility 

the  effect  of  interfacing  with  systems  outside  [their]  control 

•  poor  and  late  definition  of  requirements 

•  managing  user  expectations  (in  terms  of  requirements) 

•  identifying  and  describing  customer  requirements 

For  the  most  part,  those  responses  citing  requirement  problems  attempt  to  point  to  specific 
causes.  As  an  example,  7  of  the  18  responses  point  to  customer/users  “dictating”  and 
“redefining”  requirements  or  changing  requirements  during  system  development.  Others 
address  changes  due  to  advancing  technology.  There  are  also  references  to  changes  brought 
about  by  the  existing  DoD  acquisition  environment  and  funding  volatility  in  programmatic 
requirements.  Of  course,  all  these  types  of  requirement  changes  subsequently  impact  the 
software  requirements  for  software  intensive  systems. 


3.  See  questions  1  and  2  on  page  9  of  the  questionnaire  reproduced  in  the  Appendix. 
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If  we  look  at  the  corresponding  responses  to  the  question  of  organizational  performance  in  this 
area,  we  see  that  the  rating  of  their  organization’s  performance  was  generally  good  to 
excellent.  These  responses  seemingly  conflict  with  the  responses  to  the  open-ended  question. 
However,  the  responses  are  justified  somewhat  if  we  also  examine  who  the  organization 
depends  on  for  expertise.  Thus,  we  may  still  expect  that  project  staffs  and  acquisition 
managers  need  skills  in  techniques  to 

•  develop  requirements,  i.e.,  translate  user  requirements  into  system  requirements 

•  manage  those  requirements  in  light  of  user  “dictated”  changes,  technology  advances, 
funding  constraints 

•  evaluate  impacts  of  these  changes  on  program  objectives,  system,  and  software 
performance 

Training  is  one  method  to  improve  these  skills. 


3.6.2  Technical  Background  of  Acquisition  Managers  and 
Staff 

The  following  are  aggregated  quotes  (from  a  total  of  nine  responses)  in  the  area  of  technical 
competency  of  managers  and  their  staffs  in  managing  the  acquisition  of  software  intensive 
systems: 

•  Managers  of  software  intensive  systems  are  chosen  for  their  acquisition  experience.  They 
often  lack  any  background  in  software  and  information  technology  issues. 

•  Program  managers  rely  heavily  on  software  contractors  to  meet  program  baselines,  often 
with  disastrous  results.  PMs  allow  this  to  happen  because  they  and  their  staffs  lack 
software  management  skills  to  verfiy  contractor  progress. 

•  PMs,  PEOs,  acquisition  executives,  and  others  in  DoD  acquisition  management  do  not 
understand  software  acquisition. 

•  [Lack  of]  qualified  (educated  and  experienced)  government  personnel  to  monitor 
contracts. 

•  Failure  of  many  mangers  to  understand  software  is  not  inherently  different  than 
development  and  maintaining  hardware....  [S]oftware  developers  encourage  the  mystique 
about  software  that  discourages  many  managers. 

•  [Obtain]  a  PM  with  software  experience. 

These  responses  are  consistent  with  those  to  the  closed  ended  question  about  the  requisite 
expertise  senior  management  needs  for  such  acquisitions  (see  Section  3.4.2).  Indications  are 
that  senior  and  middle  level  managers  need  broad  knowledge  of  software  engineering 
principles.  The  depth  of  this  knowledge  should  be  such  that  these  managers 
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•  can  “de-mystify”  software  development 

•  understand  the  similarities  and  differences  between  hardware  and  software  development 

There  were  also  responses  that  described  the  need  for  senior  and  middle  level  managers 
needing  a  more  in-depth  knowledge  of  software  acquisition  management.  The  areas  of 
knowledge  should  include  the  key  competency  areas  noted  in  the  survey.  The  depth  of  this 
knowledge  should  be  sufficient  to  allow  these  managers 

•  to  establish  acquisition  strategies  that  include  software  within  the  context  of  the  system 
acquisition 

•  to  make  informed  decisions  about 

how  to  select  qualified  contractors 
how  to  manage  risks  that  impact  software 

how  to  make  trade-offs  during  contract  performance  period  to  ensure  that  software 
functionality,  performance,  and  quality  are  managed. 

While  the  responses  to  the  open-ended  questions  appear  much  more  negative  than  closed- 
ended  questions  in  these  areas,  one  may  reasonably  conclude  that  senior  level  management 
should  have  a  basic  knowledge  of  software  engineering,  knowledge  of  software  acquisition 
management  and  associated  issues,  and  a  substantial  knowledge  of  how  software  acquisition 
fits  into  a  system  acquisition  context. 

The  lack  of  knowledge  and  experience  of  PMs  and/or  their  staffs  is  confirmed  by  responses  to 
the  question  of  sources  of  software  expertise.  Here,  many  organizations  reportedly  depended 
upon  the  contractors  who  were  developing  the  systems,  rather  than  the  in-house  staff  or  the 
PMs  themselves. 

The  specific  knowledge  needed  spans  software  acquisition  management  key  competency  areas 
to  varying  degrees  as  expressed  in  the  responses  to  the  key  comptency  areas-oriented 
questions.  The  responses  to  the  acquisition  organization’s  performance  were  mostly  adequate 
to  good.  Of  course,  these  ratings  were  from  the  PEOs  and  PMs  answering  the  questionnaire. 
These  responses  do  not  track  well  with  the  need  for  senior  management  to  have  software 
expertise  at  any  level  of  depth.  It  is  possible  that  the  respondents  did  not  have  enough  expertise 
to  rate  the  performance  of  the  organization  in  these  areas. 


3.6.3  Insufficient  Resources 

An  equal  number  of  responses  (nine)  to  those  on  technical  background  of  managers  focused 
on  the  lack  or  insufficiency  of  resources.  Representative  statements  of  these  concerns  include 
the  following: 

•  limited  resources 
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•  downsizing — not  enough  people  to  do  the  job  right 

•  lack  of  time:  with  fewer  people  to  do  more  work  we  don’t  have  the  time  to  train  the 
decreasing  workforce;  we’re  relying  on  contractors  more  and  more  and  they  don’t 
necessarily  have  the  right  experience 

•  obtaining  qualified  contract  personnel  in  an  ever-changing  environment 

•  limited  resources  [in  general] 

These  comments  have  direct  “correlation”  to  the  responses  covering  training/experience  of 
managers  and  other  staff  members  as  well  as  to  those  involving  the  volatility  of  the 
environment  in  funding,  personnel,  and  requirements. 
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4  Observations  and  Recommendations 


4.1  Review 

This  study  was  conducted  based  in  part  on  the  recommendations  in  [Cavanaugh  98]  which 
asked  that  the  DoD  perform  a  needs  analysis  of  acquisition  organizations  to  elicit  their 
education  and  training  requirements  in  terms  of  skill  levels,  skill  sets,  and  delivery  methods 
for  software  acquisition  management.  As  discussed  below,  several  observations  and  findings 
in  that  report  are  supported  by  the  responses  to  the  questionnaire  used  in  this  survey. 


4.1.1  Capability  Needs 

In  general,  the  questionnaire  responses  indicated  that  the  senior  acquisition  managers  who 
participated  in  our  study  were  reasonably  well  satisfied  with  the  capabilities  of  their 
organizations  to  acquire  software  intensive  systems  that  met  their  cost,  schedule,  and 
performance  goals.  That  said,  they  also  recognized  that  there  is  room  for  improvement.  Here, 
about  40%  expressed  reservations  about  their  organizations’ capabilities  to  properly  handle 
risk  management  and  cost  and  schedule  estimation.  Other  organizational  capabilities  about 
which  over  one  quarter  of  the  PEOs  and  PMs  expressed  concern  include  software  reuse, 
security,  and  keeping  up  with  changing  software  engineering  methodologies  and  emerging 
technologies.  Several  measurement  and  evaluation  related  categories  also  were  among  those 
key  competency  areas  that  were  cited  most  often  as  having  been  adequate  at  best  (Figures  4 
and  5). 

In  addition,  the  respondents  placed  heavy  emphasis  on  understanding  software  engineering, 
software  acquisition  management,  and  the  capability  to  perform  requirements  development 
and,  especially,  requirements  management.  In  the  last  case,  “management”  related  to  a 
spectrum  of  issues  surrounding  requirements.  These  issues  included  frequent  customer 
changes,  program/funding  volatility,  and  technology;  all  contributed  to  constant  requirement 
changes  that  the  PMs  had  to  manage.  It  seemed  evident  that  skills  needed  to  deal  with  these 
problems  must  be  improved. 

Another  concern  expressed  by  the  respondents  was  the  need  for  improved  skills  covering 
software  acquisition  management  to  compensate  for  reductions  in  the  government  workforce. 
More  and  more  government  acquisition  organizations,  represented  by  the  participants  in  the 
survey,  are  relying  on  contractor  support  (such  as  federally  funded  research  and  development 
centers  [FFRDCs]  or  system  engineering  technical  assistance  [SETA]  contractors)  to  help  in 
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their  software-intensive  system  acquisitions.  In  this  situation,  the  acquisition  organizations 
must  have  sufficient  skills  to  be  able  to  obtain  the  qualified  sources  of  such  expertise.  Some 
level  of  expertise  in  software  acquisition  management  is  needed  by  the  acquisition  managers 
or  acquisition  offices  to  ensure  the  skill  levels  of  such  contractor  support  is  acquired  and 
properly  managed. 

A  majority  said  that  there  probably  “should  be  a  separately  defined  career  path  for  specialists 
in  software  acquisition  management”  (Figure  15).  Cavanaugh  points  out  that  there  are 
conflicts  in  DoD  education  and  training  documents  regarding  the  need  or  desire  to  establish  a 
separate  career  field  for  software  acquisition  management.  The  difference  may  be  that  our 
responses  are  from  the  people  who  need  the  capability,  rather  than  from  the  Office  of  the 
Secretary  of  Defense  (OSD). 

Over  90%  agreed  that  “courses  or  comparable  experience  in  software  related  management  and 
technical  topics  [should]  be  required  for  all  members  of  the  acquisition  corps  who  work  on  the 
acquisition  of  software  intensive  systems.” 


4.1.2  Training  Improvement 

The  PEOs  and  PMs  who  completed  our  survey  saw  substantial  room  for  improvement  in  DoD 
education  and  training  with  respect  to  the  management  of  software-intensive  system 
acquisitions.  While  over  half  of  them  characterized  DoD  education  and  training  courses  as 
doing  a  good  or  better  job  in  preparing  their  organizations  for  managing  the  software  related 
aspects  of  their  acquisitions,  a  large  minority  (46%)  said  it  was  adequate  or  worse  (Figure  14). 
This  compares  with  [Cavanaugh  98]  key  findings  that  state  that  the  primary  DoD  software 
acquisition  management  courses 

•  do  not  provide  sufficient  informational  content  to  cover  a  majority  of  the  selected  topic 
areas  in  this  review,  or  those  in  the  Software  Acquisition  Management  Review  Team 
(SAMERT)  key  competency  areas  [SAM  94,  SAM  96] 

•  do  not  provide  sufficient  informational  content  to  achieve  recommended  Bloom's 
Taxonomy  levels  as  described  in  the  SAMERT  Report4 

•  do  not  incorporate  effective  practices  with  respect  to  key  competency  areas  in  sufficient 
depth  to  explain  how  to  accomplish  software  acquisition  management 

Cavanaugh  notes  that  the  required  software  acquisition  management  training  from  the  Defense 
Acquisition  University  (DAU)  may  not  be  available  to  the  personnel  responsible  for  software 
acquisition  management  for  projects  (e.g.,  SETA  contractors).  This  is  a  concern  because  many 


4.  The  SAMERT  Report  [SAM  94]  details  Bloom’s  taxonomy  levels. 
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of  the  respondents  here  relied  upon  such  support  contractors  to  provide  the  necessary  software 
engineering  and  software  acquisition  management  skills  for  their  acquisitions. 

In  response  to  a  more  pointed  question  about  how  well  DoD  education  and  training  conveyed 
detailed  information  about  how  to  perform  specific  software  related  tasks,  over  60  percent  said 
that  the  situation  was  only  adequate  at  best.  Cavanaugh  points  out  that  information  currently 
available  on  effective  practices  (i.e.,  “how-to”  methods)  to  implement  and  take  advantage  of 
acquisition  reform  initiatives  is  not  sufficient  to  help  acquisition  managers,  specifically  in  the 
software  acquisition  management  area. 


4.1.3  Training  Delivery  Improvement 

A  large  number  of  respondents,  just  over  50%,  described  the  training  opportunities  provided 
by  their  organizations  as  being  poor  or  adequate  at  best.  Many  respondents  were  considering 
relying  more  on  various  distance  learning  methods  (Figure  19),  just  in  time  training  (Figure 
21),  and  commercial  off-the-shelf  courses  (Figure  20).  However,  while  there  was  clear 
interest  in  alternative  delivery  mechanisms,  the  PEOs  and  PMs  also  intended  to  continue  to 
rely  on  DoD  course  offerings  (Figure  22). 

As  a  potential  solution,  Cavanaugh  points  out  that  “currently,  the  use  of  alternative  delivery 
methods  for  educating  and  training  the  acquisition  workforce  is  not  fully  exploited  by  DAU 
[OUSD  97b].  (However,  a  concept  plan  and  an  implementation  plan  do  exist  to  implement  a 
technology-based  education  and  training  program  within  DAU  [DAU  97a,  DAU  97b].)” 


4.2  Recommendations 

The  responses  to  the  questionnaire  suggest  the  following  recommendations. 

•  Reassess  the  current  Acquisition  Career  Development  Program  certification  requirements 
to  provide  a  more  focused  and  structured  career  management  curriculum  with  defined  skill 
sets  and  minimum  technical  competencies  related  to  software  acquisition  management. 
This  reassessment  may  also  identify  the  need  for  a  separate  software  acquisition 
management  career  field  as  part  of  Acquisition  Career  Development  Program 
certification. 

•  Conduct  a  study  to  achieve  the  following  goals: 

Determine  the  best  means  of  providing  the  in-depth  knowledge,  including  both 
“how  to’s”  and  lessons  learned,  required  for  achieving  the  higher  levels  of  Bloom's 
Taxonomy  [SAM  94,  SAM  96]  as  well  as  the  advanced  levels  of  software  acquisition 
management  and  continuing  education.  At  a  minimum,  this  study  should  consider 
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alternative  delivery  mechanisms,  just-in-time  training,  long-term  certificate  programs, 
and  specialized  training  focusing  on  specific  aspects  of  software  acquisition 
management. 

Analyze  alternative  delivery  mechanisms  to  determine  which  would  best  fulfill 
software  acquisition  management’s  need  for  ensuring  that  workforce  training  is  kept 
up  to  date,  providing  on-site  training,  and  updating  already  certified  professionals. 

•  As  in  Cavanaugh,  we  suggest  that  DoD  conduct  a  feasibility  study  of  providing  specific 
skills  levels  for  software  acquisition  managers  by  training  both  software  engineering  and 
software  acquisition  management  in  an  integrated,  long-term  certification  program.  One 
way  of  achieving  this  cross-discipline  competency  is  by  training  both  software 
engineering  and  software  acquisition  in  an  integrated  environment.  (The  concept  of  an 
integrated  environment  has  already  been  applied  at  the  Air  Force  Institute  of  Technology 
in  its  Software  Professional  Development  Program. ) 

•  Finally,  we  have  no  comparable  evidence  about  the  judgments  of  less  senior  acquisition 
personnel.  The  study  does  provide  insight  and  identify  issues  from  the  perspective  of 
PEOs  and  PMs,  but  it  is  not  necessarily  generalizable  to  the  wider  DoD  acquisition 
community.  Follow-on  surveys  of  less  senior  personnel  should  be  conducted  to  provide 
additional  insights  into  problem  areas  and  suggest  potential  solutions  to  software-intensive 
systems  acquisition  management  needs. 


28 


CMU/SEI-2000-TR-003 


Appendix 


Questionnaire 


A  copy  of  the  survey  questionnaire  follows. 
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Acquisition  of  Software  Intensive  Systems 
in  the  United  States  Government 


Purpose  This  document  contains  questions  about  your  experiences  in  acquiring  software 

intensive  systems  for  use  in  DoD  organizations  and  other  federal  agencies.  The 
results  will  be  used  to  help  formulate  plans  for  education  and  training  to  improve 
the  acquisition  of  software-intensive  systems.  Your  answers  will  be  invaluable  to 
help  us  better  understand  the  needs  of  the  acquisition  workforce. 

Instructions  You  are  part  of  a  carefully  chosen  sample.  It  is  extremely  important  that  we 
receive  your  candid  and  personal  answers  in  order  for  the  results  to  be  accurate 
and  useful. 

The  questionnaire  should  take  about  thirty  minutes  of  your  time.  Please  complete 
it  as  soon  as  you  can  --  right  away  if  you  can  make  the  time. 

Confidentiality  Your  answers  will  be  held  in  the  strictest  of  confidence.  Information  that  can 
identify  you  and  your  organization  will  be  used  for  administrative  purposes  of  this 
study  only.  Your  answers  will  be  used  in  summary  statistical  form.  Specific 
answers  will  never  be  identified  by  organization  or  individual. 


Thank  you  for  your  help. 


Software  Engineering  Institute 
Carnegie  Mellon  University 
acquisition  @  sei  .emu  .edu 
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Your  Background  in  the  Acquisition  of  Software  Intensive  Systems 


•  By  “software  intensive  systems”  we  mean  systems  where  some,  though  not  necessarily 
all,  functions  of  the  system  are  implemented  through  or  rely  on  software  --  AIS,  weapons 
systems,  C3EEW. 

•  By  “acquisition”  we  mean  the  process  of  obtaining  through  contract.  Contracts  (formal  or 
informal)  are  binding  agreements  between  two  or  more  parties  that  establish  the 
requirements  for  the  products  or  services  to  be  acquired.  Acquisition  starts  with  planning 
of  the  system  to  be  acquired  and  continues  through  final  delivery  of  products  and  services. 

•  By  “software  acquisition  management”  we  mean  the  control  and  direction  of  the  technical 
and  management  decisions  that  affect  the  acquisition  of  the  software  in  acquisitions  of 
software-intensive  systems. 


1  First  of  all,  what  kind  of  software  intensive  systems  have  you  participated  in  acquiring? 

(Please  select  as  many  as  apply) 

O  AUTOMATED  INFORMATION  SYSTEMS  -  e.g.,  management  information  systems  supporting 
business  operations  such  as  payroll,  inventory,  or  logistics 

O  WEAPONS  SYSTEMS  -  e.g.,  with  real-time  process  control  or  guidance  systems  for  avionics  or 
radar;  embedded  software  running  in  electronic  devices,  vehicles,  missiles  or  air  craft 

O  C3IEW  --  e.g.,  decision  support  systems,  mission  planning,  communications  systems,  or  maneuver 
control 

O  OTHER  -  e.g.,  simulations,  compilers,  configuration  management  tools,  cost  estimation  tools, 
personal  computer  applications,  pattern  recognition,  expert  systems  (Please  describe  briefly) 


2  How  well  has  your  previous  experience  or  training  prepared  you  for  your  current  work  in  the 
acquisition  of  software  intensive  systems?  (Please  select  one) 

□  EXTREMELY  WELL 

□  VERY  WELL 

□  MODERATELY  WELL 

□  NOT  VERY  WELL 
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2.1  What  specifically  in  your  previous  experience  or  training  best  prepared  you  for  your  current 
work  (e.g.,  job  experience,  particularly  good  training  courses,  mentoring,  previous  work  on 
software  acquisitions)?  ( Please  describe  briefly) 


2.2  In  what  ways  and  in  what  areas  could  you  have  been  better  prepared  for  your  current  work  in  the 
acquisition  of  software  intensive  systems  (e.g.,  better  training  or  mentoring,  more  background  in 
software  engineering)?  ( Please  describe  briefly) 


3  How  would  you  describe  your  personal  expertise  in  ...?  (Please  select  one  for  each) 


& 


<o 


/ 


<$r 

/ 


N/ 


3.1  Technical  aspects  of  software  engineering .  O  O  O  LI 

3.2  Management  of  software  acquisitions .  0  0  0  0 

3.3  Management  of  the  acquisition  of  software  intensive  systems .  d  [1  D  D 
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About  Your  Acquisition  Organization 

•  By  “acquisition  organization”  we  mean  a  Program  Management  Office  or  other  entity  that 
has  the  oversight  responsibility  for  one  or  more  acquisitions  of  software  intensive  systems. 

•  When  thinking  about  vour  organization,  please  answer  for  the  acquisition  organization(s) 
that  you  actually  work  in  or  that  you  support  -  not  for  the  larger  entity  of  which  it  may  be  a 
part. 


1 


Following  is  a  list  of  areas  related  to  the  management  of  software  acquisition  that  may  impact  an 
organization’s  ability  to  successfully  acquire  software  intensive  systems.  How  would  you  rate  the 
capability  of  your  acquisition  organization  in  each  area?  Or  is  that  capability  unnecessary  in  your 
organization?  (Please  select  the  best  answer  for  each ) 


1.1  Developing  system  acquisition  strategies  that  address  software 

issues  (e.g.,  strengths  and  weaknesses  of  current  strategies,  impact  on 
project  planning  and  engineering  methods) . 

1.2  Incorporating  software  architecture  concepts  into  acquisition 

RFP’s  and  contracts  (e.g.,  relationships  of  software  to  system 
architectures,  and  architecture  to  software  design) . 


//  / 
///// 

/ 

/ 

□  □  □  □  □ 

n 

□  □  □  □  □ 

o 

1.3  Handling  contracting  related  documentation  (e.g.,  work  break-down 
structure  for  software,  laws  and  regulations  related  to  SOW  and  RFP, 

data  and  intellectual  property  rights,  commercial  and  DoD  best  practices)0  O  O  O  O 

1.4  Understanding  configuration  management  processes  and 

practices  (e.g.,  synchronization  of  hardware  and  software  baselines) .  LJ  LJ  LJ  LJ  LJ 

1.5  Making  estimates  of  software  costs  and  schedules  (e.g.,  strengths 
and  weaknesses  of  estimation  methods  and  models,  life  cycle  cost  and 

schedule  reporting,  validation/assessment  of  fidelity  of  cost  estimates)  LJ  LJ  LJ  LJ  LJ 

1.6  Understanding  software  issues  in  program/project  office 
organization  and  relationships  (e.g.,  staffing,  matrix  support  groups, 
resource  management,  project  control  and  tracking,  end  user 

involvement,  intergroup  coordination,  corrective  actions) .  .  □  □  □  n  □ 


□ 

□ 

n 

□ 


1.7 

1.8 


1.9 

1.10 

1.11 

1.12 

1.13 


Assessing  maturity  of  software  acquisition  and  development 

processes .  LJ  LJ 

Understanding  the  latest  software  engineering  approaches  and 
methodologies  and  incorporate  them  into  the  acquisition  (e.g., 
strengths  and  weaknesses  of  functional,  object  oriented,  and  other  current, 
approaches  effect  of  design  approach  on  software  and  system 
engineering,  CASE  selection  and  use) .  LJ  LJ 

Understanding  concepts  of  independent  verification  and  validation 
and  incorporate  them  into  the  management  of  the  software 

acquisition .  □  □ 

Understanding  life  cycle  management  (e.g.,  as  it  is  affected  by  areas 
such  as  DoD  life  cycle  guidance,  outsourcing,  and  post  deployment 

software  support) .  LJ  LJ 

Determining  how  and  when  to  use  metrics  (e.g.,  to  gain  visibility 

into  software  and  system  development  processes  and  products) .  LJ  LJ 

Incorporating  Open  Systems  concepts  and  practices  into  the 
management  of  the  acquisition .  □  □ 

Incorporating  software  quality  management  concepts  and  practices 
into  the  acquisition  (e.g.,  software  quality  attributes,  clean  room,  peer 
reviews,  software  quality  assurance  planning  and  techniques) .  LJ  LJ 


nan 

ana 

ana 

n  n  a 
nan 
ana 

ana 


a 


a 

n 

a 

a 

a 

a 
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1  (continued)  How  would  you  rate  the  capability  of  vour  acquisition  organization  in  each  area?  Or  is  that 


capability  unnecessary  in  your  organization?  (Please  select  the  best  answer  for  each) 


0  ^  <r 

/ 

1.14 

Managing  software  requirements  (e.g.,  requirements  elicitation/ 
definition,  user  involvement,  baselines  and  traceability,  critical 
measures  of  effectiveness,  managing  changing  requirements) . 

□  □  □  □  □ 

□ 

1.15 

Incorporating  software  reviews  and  audits  into  the  acquisition 
(e.g.,  critical  software  life  cycle  reviews,  relationship  of  software  and 
system  reviews) . 

□  □  □  □  □ 

□ 

1.16 

Incorporating  reusable  software  into  system  development  (e.g., 
considering  cost  factors,  software  support  transition  issues, 
outsourcing,  and  post  deployment  software  support) . 

□  □  □  □  □ 

□ 

1.17 

Performing  software  acquisition  risk  management  for  the 
acquisition  organization . 

□  □  □  □  □ 

□ 

1.18 

Incorporating  software  security  concepts  and  practices  into  the 
management  of  the  acquisition . 

□  □  □  □  □ 

□ 

1.19 

Understanding  emerging  issues  and  technologies  that  affect 
software  (e.g.,  Joint  Technical  Architecture,  domain  and  product  line 
engineering,  state  of  the  art  of  software  technology,  interoperability 
issues) . 

□  □  □  n  □ 

□ 

1.20 

Evaluating  software  related  processes,  products,  and  services  to 
determine  if  contractual  requirements  are  being  satisfied . 

□  □  □  □  □ 

□ 

1.21 

Providing  appropriate  training  to  personnel  who  manage 
software  acquisitions . 

□  □nan 

□ 

1.22 

Understanding  the  implications  on  the  acquisition  of  software 

intensive  systems  of  DoD  acquisition  reforms  (e.g.,  integrated 
product  and  process  teams,  electronic  commerce  and  data  interchange, 
oversight  reduction,  earned  value  management,  or  the  Defense 

Acquisition  Deskbook) .  D  D  O  D  D  O 

1.23  Are  there  any  other  important  areas  that  you  think  belong  in  this  list?  How  would  you  rate  the 
capability  of  your  acquisition  organization  in  those  areas?  (Please  describe  briefly) 
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2  Overall,  how  would  you  rate  the  performance  of  your  organization  in  acquiring  the  software  for  the 
systems  you  acquire  (e.g.,  meeting  schedule  and  budget  commitments,  and  the  technical  performance 
once  deployed  of  the  software)?  (Please  select  one  for  each) 

□  EXCEPTIONAL 

□  EXCELLENT 

□  GOOD 

□  ADEQUATE 

□  POOR 


3 


On  whom  does  your  organization  rely  for  expertise  about  managing  the  software  related  aspects  of  its 
acquisition(s)?  (Please  select  one  for  each) 


x 

& 


*  /  /  ^ 

/$•  Ut  *v  x 
^  <§?  &  .'T 

cf  ^ 

■  ~  /p  ^ 


/? 


3.1  In-house  staff  in  the  acquisition  organization 


□  □  □  n 


3.2  Other  government  organizations  that  are  used  to  support  the 
organization’s  software  intensive  acquisitions . 


□  □  □  □ 


3.3  The  industry  contractors  from  whom  the  organization 

acquires  software  intensive  systems .  O 

3.4  Other  contractors  that  provide  direct  support  to  the  organization ..  d 


□  n  n 

□  □  □ 


4  What  level  of  expertise  do  you  think  senior  management  needs  in  ...  ? 
(Please  select  the  best  answer  for  each) 


4.1  Technical  aspects  of  software  engineering . 

4.2  Management  of  software  acquisitions . 

4.3  Management  of  the  acquisition  of  software  intensive  systems 
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□  □  □  □ 

□  n  □  n 

□  □  □  □ 
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Training  Issues 

1  In  general,  how  good  a  job  do  you  think  current  DoD  education  and  training  does  in  preparing  your 
organization  for  managing  the  software  related  aspects  of  your  acquisitions?  (Please  select  one) 

□  EXCEPTIONAL 

□  EXCELLENT 

□  GOOD 

□  ADEQUATE 

□  POOR 


2  How  well  do  you  think  current  DoD  education  and  training  convey  software  related  skills  --  with 
sufficient  detail  and  direction  about  how  to  perform  necessary  tasks?  (Please  select  one) 

O  EXCEPTIONAL 

□  EXCELLENT 

□  GOOD 

□  ADEQUATE 

□  POOR 


3  Do  you  think  that  there  should  be  a  separately  defined  career  path  for  specialists  in  software 
acquisition  management?  (Please  select  one) 

□  DEFINITELY 

□  PROBABLY 

□  PROBABLY  NOT 

□  DEFINITELY  NOT 

□  DON'T  KNOW 


4  Should  courses  or  comparable  experience  in  software  related  management  and  technical  topics  be 
required  for  all  members  of  the  acquisition  corps  who  work  on  the  acquisition  of  software  intensive 
systems?  (Please  select  one) 

□  DEFINITELY 

□  PROBABLY 

□  PROBABLY  NOT 

□  DEFINITELY  NOT 

□  DON'T  KNOW 
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5  Which  of  the  following  does  your  organization  currently  rely  on  to  prepare  personnel  to  manage  the 
acquisition  of  software  intensive  systems?  What  ought  you  rely  on  ...  ?  ( Please  select  one  for  each) 

Currently  ...  Ought  to  ... 


5.1 

5.2 

5.3 

5.4 


5.5 

5.6 


5.7 


Formal  education  and  pre-employment  training 
In-service  continuing  education  and  short  courses 


i3 

^  * 


r/,‘ 


-sF  4*“  &  <? 


□  □  □  □ 
□  n  □  n 


A 

vy  & 
4 


£ 


On  the  job  training  and  developmental 

assignments .  □ 

“Distance  learning”  methods  (e.g.,  web  based,  CD 
ROMs,  multimedia,  satellite  broadcasts,  network  and 
computer  based  instruction,  collaborative  groupware)  O 

Focused  “just  in  time”  training .  O 

Commercial  off-the-shelf  courses  to  augment  or 
replace  training  provided  directly  by  the  DoD 
and  other  government  agencies .  D 

Existing  DoD  courses .  O 


□  □  □ 


□  □  □ 
□  □  □ 


n  n  n 

ODD 


&  ^ 


X 


&  $  i 


/ 

<</ 


<?  <? 


□  n  □  □ 

□  a  □  a 


□  □  □  □ 


□  □  □  □ 
naan 


□  n  □  □ 

□  n  □  □ 


6 


Following  is  a  list  of  statements  that  are  sometimes  made  about  an  organization’s  ability  to  obtain 
adequate  training  about  software  related  aspects  of  system  acquisition.  How  well  do  these  statements 
apply  to  vour  organization?  ( Please  select  one  for  each) 


6.1 

6.2 

6.3 

6.4 

6.5 

6.6 

6.7 

6.8 

6.9 

6.10 

6.11 


Key  staff  rotate  out  of  the  organization  at  inopportune  times  before 

the  acquisition  is  finished . 

There  aren’t  enough  good  training  courses  available  to  properly 
prepare  our  staff  for  their  software  acquisition  management  duties 
Existing  courses  are  too  long  and  demanding.  Our  staff  can’t 

spare  the  time  to  attend  them . 

Exisiting  courses  are  too  cursory.  They’re  not  not  long  enough  to 

convey  usable  information  or  help  people  develop  new  skills . 

We  don’t  have  enough  travel  or  other  discretionary  money 
available  to  send  people  for  the  training  they  need . 

We  sometimes  send  the  wrong  people  for  training . 

The  organization  has  lost  critical  staff  to  better  paying  jobs  in 

industry . 

The  organization  has  lost  critical  staff  to  downsizing  of  the 

acquisition  workforce . 

We  can’t  afford  the  time  to  send  key  personnel  to  training  courses; 

we  need  to  keep  them  on-site  and  on  the  job . 

The  organization  doesn’t  spend  enough  time  bringing  new  staff 
up  to  speed.  Critical  time  and  resources  are  wasted  rediscovering 

what  is  already  well  known  and  understood . 

Despite  the  best  of  intentions,  defined  processes  and  procedures 

are  overcome  by  events  and  crises.  Other  things  take  priority . 
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Problems  Faced  in  Acquiring  Software  Intensive  Systems 


1  Overall,  what  are  the  one  or  two  most  difficult  problems  that  your  organization  has  faced  in  conducting 
successful  acquisitions  of  software  intensive  systems?  (Please  describe  fully) 


2  If  you  could  change  one  or  two  things  about  how  software  intensive  systems  are  acquired  in  the  DoD, 
what  would  they  be?  (Please  describe  fully) 
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Acquisition  of  Software  Intensive  Systems 
in  the  United  States  Government 


Please  return  this  form  at  your  earliest  convenience 
Use  the  enclosed  envelope,  or  send  it  to: 

Robert  Rosenstein 

I  __ 

Software  Engineering  Institute 
4500  Fifth  Avenue,  5th  Floor 
Pittsburgh,  PA  15213-2691 
412/268-8468 
rrosenst@sei.cmu.edu 
fax:  412/268-5758 

L 
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